Skip to content

CI: harden build-wheel.yml for the eMudhra/emSign CA incident (12.9.x backport) - #3306

Draft
juenglin wants to merge 1 commit into
NVIDIA:12.9.xfrom
juenglin:emudhra-build-wheel-12.9.x
Draft

juenglin wants to merge 1 commit into
NVIDIA:12.9.xfrom
juenglin:emudhra-build-wheel-12.9.x

Conversation

@juenglin

Copy link
Copy Markdown
Contributor

Description

Refs #3195

Backports the CI hardening that main received for the eMudhra/emSign CA key compromise (CVE-2026-111943, CVE-2026-115557) to 12.9.x. The cuda-bindings wheels built here are also consumed by main, whose previous-CUDA-major cuda-core wheel is built against the latest 12.9.x CI run, so this branch's build workflow affects main releases as well.

  • upgrade-ca-certificates action: first step of build-wheel.yml (CI: add eMudhra/emSign CA stopgap for GitHub-hosted Windows runners #3270).
  • setup-hardened-python action: replaces actions/setup-python plus harden-pip in build-wheel.yml and test-wheel-{linux,windows}.yml (CI: install eMudhra-fixed Pythons with a pinned uv #3275). This branch still builds and tests Python 3.10, which has no fixed release, so for series older than 3.11 the action warns and installs the sha256-pinned pip 26.2.2 wheel instead of failing. The Python include path is derived from sysconfig because uv interpreters are not at Python3_ROOT_DIR. cibuildwheel now runs with the hardened host Python (python-path).
  • cibuildwheel: install the sha256-pinned pip 26.2.2 in CIBW_BEFORE_BUILD_LINUX and CIBW_BEFORE_BUILD_WINDOWS before anything else is downloaded. Linux had no before-build step and Windows ran an unpinned pip install delvewheel.
  • fetch_ctk: verify the CTK redistrib JSON against a pinned sha256 and each component archive against the sha256 in that JSON before extracting it, failing closed on a mismatch (new ci/tools/fetch_ctk_redistrib.py, with unit tests). The curl download no longer passes -k, which disabled TLS verification. Pinned versions: 11.8.0, 12.0.1, 12.6.3, 12.9.1, 13.0.1, 13.0.2 and 13.4.2; unpinned versions only warn.
  • install_gpu_driver.ps1: verify the fixed Windows driver installer against a pinned sha256 before running it as admin (CI: verify GPU driver installer SHA-256 before execution #3249, for this branch's single installer).

This branch has no sccache and no Linux driver install script, so install-sccache and the Linux part of #3249 do not apply. Not backported: the runner trust probe and the "Enable Git symlinks" steps.

Notes for reviewers

The harden-pip action is removed because setup-hardened-python subsumes it. This branch keeps the pip frontend in cibuildwheel and fixes pip instead of switching to build[uv].

Validation: the new unit tests pass locally, the pre-commit hooks pass, and the pinned redistrib hashes were re-checked against the live JSONs (the 12.9.1 hash matches the one pinned on 13.4.x). The driver installer hash was computed from two independent downloads that agree. The workflow changes have not run on GitHub Actions yet, so CI on this PR is the real test, in particular the Windows jobs and the Python 3.10 fallback.

Checklist

  • New or existing tests cover these changes.
  • The documentation is up to date with these changes.

… backport)

Backport the CI hardening that main received for the eMudhra/emSign CA key
compromise (CVE-2026-111943, CVE-2026-115557), so that wheels released from
this branch, and the cuda-core wheels on main that are built against this
branch's cuda-bindings wheels, are built with patched tooling and verified
downloads:

- upgrade-ca-certificates: upgrade ca-certificates on GitHub-hosted Linux
  and distrust the compromised intermediates on GitHub-hosted Windows, as
  the first step of build-wheel.yml (NVIDIA#3270).
- setup-hardened-python: install interpreters with a pinned uv and fail on
  unpatched ones, replacing actions/setup-python + harden-pip in
  build-wheel.yml and test-wheel-{linux,windows}.yml (NVIDIA#3275). This branch
  still builds and tests Python 3.10, which has no fixed release; for series
  older than 3.11 the action warns and installs pip from the sha256-pinned
  26.2.2 wheel instead of failing. cibuildwheel now runs with the hardened
  host Python (`python-path`), and the Python include path is derived from
  sysconfig because uv interpreters are not at Python3_ROOT_DIR.
- cibuildwheel: install the sha256-pinned pip 26.2.2 in
  CIBW_BEFORE_BUILD_LINUX and CIBW_BEFORE_BUILD_WINDOWS before anything else
  is downloaded (Linux had no before-build step; Windows ran an unpinned
  `pip install delvewheel`).
- fetch_ctk: verify the CTK redistrib JSON against a pinned sha256 and every
  component archive against the sha256 in that JSON before extracting it,
  failing closed on a mismatch (ci/tools/fetch_ctk_redistrib.py, with
  tests). Also stop passing `-k` to curl, which disabled TLS verification
  for the archive downloads.
- install_gpu_driver.ps1: verify the fixed Windows driver installer against
  a pinned sha256 before running it as admin (NVIDIA#3249 for this branch's single
  installer).

This branch has no sccache and no Linux driver install script, so the
install-sccache action and the Linux part of NVIDIA#3249 do not apply. Not
backported: the runner trust probe and the "Enable Git symlinks" steps.

Refs NVIDIA#3195
@juenglin juenglin added the CI/CD CI/CD infrastructure label Oct 30, 2026
@copy-pr-bot

copy-pr-bot Bot commented Oct 30, 2026

Copy link
Copy Markdown
Contributor

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@juenglin juenglin self-assigned this Oct 30, 2026
@juenglin

Copy link
Copy Markdown
Contributor Author

/ok to test 09b94e2

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CI/CD CI/CD infrastructure

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant